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(57) A method for capturing at least one image of an 
existing man-made structure and for detecting the pres- 
ence of failure(s) in such existing man-made structure 
including: providing a detectable material on the surface 
or in an existing man-made structure so that portions of 
the detectable material will be present in the failure(s) 
of the existing man-made structure; providing an image 
sensor and which captures at least one image of the ex- 
isting man-made structure and identifies faiture(s) due 
to the existence of the detectable material in the failure 
(s) to provide at least one digital image; and processing 
the captured digital image(s) to provide a visual image 
of the existing man-made structure to determine the 
presence of failure(s) in the existing man-made struc- 
ture. 
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Description 

[0001] The present invention relates to detecting the 
presence of failure(s) in existing man-made structures. 
[0002] The surveillance of ground topography is well 5 
known in the art. It is frequently the case that an aircraft 
or a satellite includes an image capture device such as 
a charge-coupled device (CCD). In ground surveillance 
it is highly desirable to detect whether there has been a 
failure in a man-made object such as a road, a pipeline, 10 
an electrical grid, or other man-made structures of prac- 
tical interest. When detected, a determination is made 
if remedial action must be taken. Often times a visual 
inspection of ground topography is provided by a land- 
based crew that traverses an area by vehicle or foot, to 
determine if there is a failure. Airborne photographic 
systems can also be used for capturing images of adja- 
cent areas. These images are then reviewed to deter- 
mine if there is a failure. 

[0003] In many cases these inspections must done by 
a ground site survey; individuals visit these locations 
and take measurements or obtain other forms of data 
on-sight. This process becomes cumbersome, costly, 
inconvenient, and in many cases unreliable and unsafe 
due to the dangers present in remote locations and to 
potential false interpretations due to worker fatigue and 
other factors. Furthermore, remote locations are fre- 
quently in mountains, deserts and forests that are diffi- 
cult to reach and frequent inspection requires the place- 
ment of permanent maintenance and inspection crews 30 
adding to overall costs. 

[0004] It is an object of the present invention to pro- 
vide an improved way to determine if there is a failure 
in an existing man-made structure. 

[0005] It is another object of the present invention to 35 
use material that can be detected by an image sensor 
when present in a failure of an existing man-made struc- 
ture. 

[0006] This object is achieved by a method for cap- 
turing at least one image of an existing man-made struc- 40 
ture and for detecting the presence of failure(s) in such 
existing man-made structures comprising the steps of: 

(a) providing a detectable material on the surface 

or in an existing man-made structure so that por- 45 
tions of the detectable material will be present in the 
failure(s) of the existing man-made structure; 

(b) providing an image sensor and which captures 
at least one image of the existing man-made struc- 
ture and identifies failure(s) due to the existence of 50 
the detectable material in the failure(s) to provide at 
least one digital image; and 

(c) processing the captured digital image(s) to pro- r 
vide a visual image of the existing man-made struc- 
ture to determine the presence of failure(s) in the 55 
existing man-made structure. 

[0007] It is an advantage of the present invention to 



provide a more effective way of determining failure(s) in 
existing man-made structures by automatically process- 
ing images captured from a remote platform. This auto- 
matic processing can include comparing with previously 
detected images. This automatic processing can also 
include algorithms and expert systems that act in a pre- 
dictive manner. 

[0008] A feature of the present invention is that a de- 
tection chemical or agent or a chemical change associ- 
ated with a failure can be applied to an existing man- 
made structure of practical interest that is particularly 
suitable for detection after a failure occurs. 

FIG. t depicts a system for capturing images from 
an airborne, a satellite or ground platform in accord- 
ance with the present invention; 
FIG. 2 is ?. flowchart in block diagram form of the 
process of capturing and processing images to de- 
tect failures in existing man-made structures; 
FIG. 3 is a flowchart in block diagram form of an 
image-processing algorithm, which can be used in 
the system shown in FIG. 1 ; 

FIG. 4 illustrates a method using a truck of applying 
a chemical material or agent on a road to aid in de- 
tecting fa^;res in an existing man-made object; 
FIG. 5 illuvA ates another method of applying the de- 
tection material or agent in accordance with the 
present invention; and 

FIGS. 6a and 6b illustrates a method of applying 
and spreading the detection material or agent using 
a vehicle in accordance with the present invention; 
FIG. 7 illustrates a somewhat different embodiment 
of the present invention that uses the absence of a 
chemical material or agent to aid in detecting failure 
(s) in an existing man-made structure; and 
FIG. 8 illustrates another embodiment of the 
present invention that uses a chemical material or 
agent to aid in the detection of leaks in a pipeline. 

[0009] In the case where a detectable substance has 
to be applied to the structure an expense is occurred in 
applying the substance. In the case of a road the sub- 
stance could be added to the road building material itself 
such as concrete or asphalt. A problem exists with cur- 
rent roads. An inexpensive delivery method is needed 
for the detectable substance that is to be detected. For 
example, if a detection chemical material or agent pro- 
vides a different in contrast when there is a material fail- 
ure an inexpensive delivery system can be used to apply 
the detection chemical material or agent. Also it can be 
advantageous for the detectable substance to have a 
short life or a degree of biodegradability. The present 
invention can use encapsulated dyes or phosphors. 
Such dyes or phosphors can be encapsulated in a pol- 
ymeric or ceramic matrix for easier delivery. That is, for 
the substance to be present only for the time the images 
are to be captured. 

Referring to FIG. 1 , a sensor system 1 employed in the 
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capturing of images in order to identify failures in exist- 
ing man-made structures can be mounted on either an 
aerial platform 21 , a satellite 26 or a ground based plat- 
form 28. Images of the ground containing various exist- 
ing man-made structures such as roadways, pipelines, 
electrical power lines, agricultural, mining, real estate 
and the like are captured by this sensor system 1 . Se- 
quential images can be captured in digital form and ei- 
ther stored in the aerial platform 21 : satellite 26 or 
ground based platform 28 to be transferred later, or 
transmitted via a radio link to a control ground station. 
The capture device 2 includes an electronic sensor, typ- 
ically a CCD or complementary metal oxide semicon- 
ductor (CMOS) imaging array that along with some im- 
aging optics captures a picture of the scene in electronic 
form. In some instances a special optical filter 3 is at- 
tached to the input to the CCD or CMOS detector to filter 
the light wavelengths that are incident upon the detector. 
This optical filter 3 is chosen so as to maximize the sig- 
nal-to-noise ratio for the detection of a specific type of 
failure. Alternatively, the ground location image can be 
captured by conventional photographic cameras. Film 
images would then have to be converted to digital im- 
ages by an image scanner that includes an image sen- 
sor. The system 1 also has an image capture control 
circuit 4 that sequences the operation of the capture de- 
vice 2. As will be clear from FIG. 1 , the operation of the 
various elements shown in system 1 are under the con- 
trol of a control computer 31 . The image capture control 
circuit 4 controls capture device 2 and sends position 
and orientation information to a position and orientation 
storage circuit 5 with each captured image. Position in- 
formation in the form of spatial coordinates is provided 
in order to identify the location of the existing man-made 
structures of interest. Such position information is also 
stored in position and orientation storage circuit 5. Po- 
sition and orientation data are used along with prede- 
termined coordinate positions to locate the existing 
man-made structures in the captured image. Control 
computer 31 causes image data to be stored in image 
storage 6 and can be processed to identify features of 
a scene in image processing circuit 7. The processing 
sequence is also directed by control computer 31 . The 
image processing circuit 7 includes a storage memory 
(not shown) that includes a representation of different 
failures to be detected and comparing the captured dig- 
ital image with the failures to determine the presence of 
a failure(s), type of failure(s) and location of the failure 
(s). With the exception of the capture device 2, the var- 
ious elements of the system 1 can be located either in 
the remote platform or at the ground station location. 
Moreover, many of the elements described can be em- 
bodied in software that can be understood to be within 
the control computer 31 . The capture device 2 is located 
in either the aerial, satellite or ground based platform. 
[0010] The overall process for detecting failure(s) in 
existing man-made structures is depicted in flowchart 
form in FIG. 2. The flowchart is in block diagram form 
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and those skilled in the art will appreciate that many of 
the functions are controlled by the control computer 31 . 
The starting event includes initializing the capture de- 
vice 8 and image storage 6 to erase any previously cap- 
5 tured scene data. Next a new scene is captured in block 
9 using the position information supplied by the custom- 
er to trigger recording of the images. The image data 
along with position and time information necessary to 
identify the location and time of the current scene is 

*0 stored in order to facilitate comparison with the same 
scene taken at other times. Image and other data are 
stored in a scene database 10 in order to perform such 
comparisons at a future time. Image analysis 11 is next 
performed in order to identify changes in the scene and 

*5 facilitate identification of failures in the existing man- 
made structures that appear in the scene. The latest 
scene image is compared with image data that has been 
previously stored in the scene database 10. If a failure 
is not detected the process stops. Detection of a failure 

20 can initiate further image analysis 1 2 as required. The 
identification process finishes with the communication 
of the results of the analysis 13. The communication of 
the results can take many forms, for example an audible 
or visual alarm, a telephone contact ore-mail notification 

25 of the detection of the failure. The final step in the proc- 
ess is to correct the failure. 

[0011] FIG. 3 depicts the algorithm used to process 
image data files from a database and identifies failures 
if they have occurred. The algorithms can embody an 

30 expert system which permits interplay with a user. Two 
separate data files, scene (1)14 and scene (2) 15, are 
made available for comparison. Both data files contain 
the same scene content, but they typically record imag- 
es taken at different times. That is, the time between 

35 capturing the two images differs by a time At. Bothimage 
files and scenes can undergo the process of orthorecti- 
fication 16, that is, compensation for variations in posi- 
tion and angle at the time the scenes were recorded. 
This process is performed in order to allow an exact pix- 

40 el-by-pixel comparison of the elements of a scene or im- 
age. It may or may not be necessary to correct the data 
in each scene for differences in the illumination 1 7 at the 
time each scene was recorded. Changes in the scene 
are identified in block 18 are used by the control corn- 

45 puter31 by detecting, using software, differences in the 
pixel content of the two scenes to be compared. Such 
changes can be reflected in the intensity of the pixels, 
or in the shape of an object, corresponding to a finite 
collection of pixels. Such methods for identification of 

50 pixel or object changes are well known to those skilled 
in the art. On the basis of such pixel changes the failure 
type is identified in block 19 and notification 20 of the 
existence of the failure is communicated. 
[0012] FIG. 4 illustrates a method of applying the de- 

55 tection chemical material or agent to an existing struc- 
ture. In the embodiment shown the detection chemical 
material or agent 40 is mixed with salt, sand, cinders, 
etc, which is normally applied to a road 42 by a truck 44. 
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The detection chemical material or agent 40 can be en- 
capsulated fluorescent dyes mixed with road salt, sand 
or cinders so when the salt., sand or cinders is applied 
to the road 42 the dyes would be spread over the road 
surface with the salt. If water soluble dyes were used, 5 
they would readily been spread with the salt. After the 
salt has been applied, the salt tends to congregate in 
cracks 27 as shown in FIG. 5 and so would the dye those 
emphasizing areas of road wear. Yellow-emitting halo 
phosphate phosphors are well established as sensors 10 
when used in this mode of operation. 
[0013] FIG. 5 illustrates the uses of the detection 
chemical material or agent 40 to aid in detecting failures 
in the existing man-made object. An aerial platform 21 
performs image capture 22 of a man-made structure (in is 
this case a road 42) in the manner previously described. 
FIG. 5 also shows image capture of a road 42 with an 
isolated failure 24. In this instance, the detection of the 
failure is enhanced by the presence the chemical image 
contrast-enhancing detection chemical material or 20 
agent 40. The association of the detection chemical ma- 
terial or agent 40 with the failure 24 results in a more 
readily detectable physical/chemical change 24a. 
[0014] In another use scenario shown in FIG. 6a, the 
detection chemical material or agent 54 is incorporated 25 
into an existing man-made structure in a form of a tape Claims 
50 applied over the width of the road 42 where it be- 
comes activated when a vehicle 52 drives over the tape 
50. The detection chemical material or agent 54 would 
adhere to the tires 56 of the vehicle and be dispersed 
over the road's surface. Other delivery techniques are 
tire baths 58 as shown in FIG. 6b. For example, the tire 
bath 58 can be put at a toll plaza not shown. As the ve- 
hicle 52 stops to pay the toll the tires 56 pass through 
the tire bath 58 containing the detection chemical ma- 
terial or agent 54 which would then, be spread over the 
road 42 by the tires 56. In this embodiment the detection 
chemical material or agent 54 adheres to the road 42 
surface in such as way that it does not accumulate in 
cracks 27 and potholes 60 as shown in FIG. 7. The fail- 
ures 24, such as cracks 27 and potholes 60 become ev- 
ident due to the absence of the detection chemical ma- 
terial or agent 54. Such detection chemical material or 
agents can include fluorescing compounds or com- 
pounds that enhance the reflectivity of the road surface 45 
in either the visible or infrared spectral regions. Moreo- 
ver, the detection chemical material or agents 54 can be 
selected for use in the visible and invisible range of the 
spectrum. The detection chemical material or agents 54 2. 
can be a family of materials known as lanthanides, 50 
wherein the lanthanides are combined with halogen el- 
ements to form a detectable halide. Still further the de- 
tection chemical material or agents 54 can be selected 
to have cholesteric characteristics such that their orien- 
tation change based on thermal equilibrium with the sur- 55 
roundings and thereby exhibit a different color that can 3. 
be detected by an appropriately selected sensor. 
[0015] In another embodiment of the present inven- 



tion as illustrated in FIG. 8, the detection chemical ma- 
terial oragent 70 is added to the material flowing through 
a pipeline 80 held up by supports 84. The detection 
chemical material or agent 70 can be provided in a liquid 
or solid solution, but must be selected to be flowable in 
the pipeline 80. For example by adding the detection 
chemical material or agent 70 to the oil a failure 24 such 
as a crack 27, which causes a leak in or spill 82 from 
the pipeline 80, can easily be detected and recorded re- 
motely. Small concentrations of oil soluble materials that 
fluoresce at visible wavelengths could significantly en- 
hance the detection of an oil leak. The use of such ma- 
terials is well known to those skilled in the art, and can 
be used singly or in combination, in remote sensing ap- 
plications. 

[0016] Such detection chemical material or agents 70 
in combination with the optical filter 3 in FIG. 1 signifi- 
cantly improve the signal-to-noise ratio, and hence the 
detectability for remote sensing. The combination of the 
optical filter transmission function and the color-change 
to be detected as a result of a failure will be optimized 
for each application. This optimization methodology is 
well known to those skilled in the art. 
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A method for capturing at least one image of an ex- 
isting man-made structure and for detecting the 
presence of failure(s) in such existing man-made 
structures comprising the steps of: 

(a) providing a detectable material on the sur- 
face or in an existing man-made structure so 
that portions of the detectable material will be 
present in the failure(s) of the existing man- 
made structure; 

(b) providing an image sensor and which cap- 
tures at least one image of the existing man- 
made structure and identifies failure(s) due to 
the existence of the detectable material in the 
failure(s) to provide at least one digital image; 
and 

(c) processing the captured digital image(s) to 
provide a visual image of the existing man- 
made structure to determine the presence of 
failure(s) in the existing man-made structure. 

The method of claim 1 wherein the image sensor is 
spaced remotely from the existing man-made struc- 
ture and further including: 

(d) sending captured processed digital images 
with detected failures to a customer 

The method of claim 1 wherein the digital image 
processing includes comparing previously captured 
digital images with newly captured digital image(s) 
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to determine variations in the captured digital image 
(s) at predetermined coordinates which indicate a 
potential failure in the existing man-made structure. 

The method according to claim 1 wherein the digital 
image(s) is captured by a capture device which is 
located in a fixed structure position above the 
ground location or in a moving structure including 
an aircraft or satellite. 



The method of claim 6 where the detectable mate- 
rial is a lanthanide, and wherein the lanthanide is 
combined with halogen elements to form a detect- 
able halide. 



w 



5. The method of claim 3 wherein the image process- 
ing includes storing in memory a representation of 
different failures to be detected and comparing the 
captured digital image with the failures to determine 

the presence of a failure, and location of such fail- 15 
ure. 

6. The method of claim 1 wherein the detectable ma- 
terial interacts with incident radiation to cause radi- 
ation from the failure in the existing man-made 20 
structure to be detected by the image sensor. 

7. The method of claim 6 wherein the detectable ma- 
terial includes encapsulated dyes or phosphors. 

25 

8. The method of claim 6 wherein the detectable ma- 
terial is included in a liquid solution or solid solution 
which is distributed on the failure in the existing 
man-made structure. 
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10. The method in claim 6 where the detectable mate- 
rial include cholesteric characteristics and change 
their orientation based on thermal equilibrium with 
the surroundings and to thereby exhibit a different 
color that can be detected by the image sensor. 40 
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